Machine-Scored Syntax: Comparison of the CLAN Automatic Scoring Program to Manual Scoring

Autor/inn/en	Roberts, Jenny A.; Altenberg, Evelyn P.; Hunter, Madison
Titel	Machine-Scored Syntax: Comparison of the CLAN Automatic Scoring Program to Manual Scoring
Quelle	In: Language, Speech, and Hearing Services in Schools, 51 (2020) 2, S.479-493 (15 Seiten) PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0161-1461
Schlagwörter	Syntax; Scoring; Computational Linguistics; Child Language; Accuracy; Comparative Analysis; Nouns; Phrase Structure; Verbs; Sentence Structure + Suchen Sie Ihr Suchwort? Bewertung; Linguistics; Computerlinguistik; 'Children''s language'; Kindersprache; Phrasenstruktur; Satzbau; Satzstruktur
Abstract	Purpose: The results of automatic machine scoring of the Index of Productive Syntax from the Computerized Language ANalysis (CLAN) tools of the Child Language Data Exchange System of TalkBank (MacWhinney, 2000) were compared to manual scoring to determine the accuracy of the machine-scored method. Method: Twenty transcripts of 10 children from archival data of the Weismer Corpus from the Child Language Data Exchange System at 30 and 42 months were examined. Measures of absolute point difference and point-to-point accuracy were compared, as well as points erroneously given and missed. Two new measures for evaluating automatic scoring of the Index of Productive Syntax were introduced: Machine Item Accuracy (MIA) and Cascade Failure Rate--these measures further analyze points erroneously given and missed. Differences in total scores, subscale scores, and individual structures were also reported. Results: Mean absolute point difference between machine and hand scoring was 3.65, point-to-point agreement was 72.6%, and MIA was 74.9%. There were large differences in subscales, with Noun Phrase and Verb Phrase subscales generally providing greater accuracy and agreement than Question/Negation and Sentence Structures subscales. There were significantly more erroneous than missed items in machine scoring, attributed to problems of mistagging of elements, imprecise search patterns, and other errors. Cascade failure resulted in an average of 4.65 points lost per transcript. Conclusions: The CLAN program showed relatively inaccurate outcomes in comparison to manual scoring on both traditional and new measures of accuracy. Recommendations for improvement of the program include accounting for second exemplar violations and applying cascaded credit, among other suggestions. It was proposed that research on machine-scored syntax routinely report accuracy measures detailing erroneous and missed scores, including MIA, so that researchers and clinicians are aware of the limitations of a machine-scoring program. (As Provided).
Anmerkungen	American Speech-Language-Hearing Association. 2200 Research Blvd #250, Rockville, MD 20850. Tel: 301-296-5700; Fax: 301-296-8580; e-mail: lshss@asha.org; Web site: http://lshss.pubs.asha.org
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01